What about Inputting Policy in Value Function: Policy Representation and Policy-Extended Value Function Approximator

We study Policy-extended Value Function Approximator (PeVFA) in Reinforcement Learning (RL), which extends conventional value function approximator (VFA) to take as input not only the state (and action) but also an explicit policy representation. Such extension enables PeVFA preserve values of multiple policies at same time and brings appealing characteristic, i.e., generalization among policies. formally analyze under Generalized Policy Iteration (GPI). From theoretical empirical lens, we show that generalized estimates offered by may have lower initial approximation error true successive policies, is expected improve consecutive during GPI. Based on above clues, introduce a new form GPI with leverages along improvement path. Moreover, propose representation learning framework for RL policy, providing several approaches learn effective embeddings from network parameters or state-action pairs. In our experiments, evaluate efficacy OpenAI Gym continuous control tasks. For representative instance algorithm implementation, Proximal Optimization (PPO) re-implemented paradigm achieves about 40% performance its vanilla counterpart most environments.